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COLLECTIVE  COMPUTATION  OF  NEURAL  NETWORK 
Yao  Guozheng  and  Wang  Yunjiu 

Institute  of  Biophysics,  Chinese  Academy  of  Sciences,  Beijing 

ABSTRACT 

Computational  neuroscience  is  a  new  branch  of  neuroscience 
originating  from  current  research  on  the  theory  of  computer 
vision.  The  goal  of  the  branch  is  to  find  a  new  inter¬ 
disciplinary  approach  to  dealing  with  information  processing  in 
the  brain.  The  research  involves  scientists  working  in 
artificial  intelligence  engineering  and  neuroscience.  The  paper 
introduces  the  collective  computational  properties  of  model 
neural  networks,  mainly  in  a  theoretical  framework  of  the  study 
of  neural  network  computational  properties  advanced  by  Hopfield 
and  its  application  to  vision  research.  On  this  basis,  the 
authors  analyzed  the  significance  of  the  Hopfield  model. 

Key  phrases:  Computational  Neuroscience,  Neural  Network, 
Model  Ciruit,  l  Dynamics  of  Neural  Network,  -  ' ssir  ‘  ? 

I .  Foreword 

The  authors  pointed  out  the  following  [1]:  with  the  central 
topic  of  visual  information  processing,  computer  vision  not  only 
creates  conditions  for  our  overall  understanding  of  human  vision 
in  biointelligence,  but  also  establishes  a  theoretical  basis  for 
developing  an  automatic  graphic  analysis  system,  thus  making 
research  on  vision  the  vanguard  in  the  investigation  of  brain 
information  processing. 
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Now  what  sort  of  computation  can  the  nervous  system  engage 
in?  In  discussing  this  kind  of  problem,  the  obviously  correct 
answer  is  to  be  expected  only  within  the  computational  theory 
framework  of  Marr  [2-3],  because  this  theory  specifies  the 
general  goal  of  computational  neuroscience.  In  other  words,  in 
dealing  with  specific  problems  that  a  neural  network  should 
solve,  not  only  should  a  researcher  decide  the  input  image  with 
the  expectation  of  the  output  form,  but  he  also  should  decide  the 
algorithm  for  execution  by  this  neural  hardware  in  computation  in 
order  to  solve  this  problem.  Hence,  computational  neuroscience 
especially  stresses  how  the  neural  hardware  executes  computations 
using  an  algorithm.  In  other  words,  this  approach  stresses  the 
properties  of  a  single  neuron,  the  synaptic  connections  between 
neurons,  the  dynamic  properties  of  the  neural  network,  and  how 
the  execution  can  lead  to  a  special  algorithm  for  execution. 

However,  often  researchers  do  not  directly  study  the 
bioneural  network;  they  study  a  model  neural  network.  The  reason 
is  very  simple.  The  bioneuron  is  a  continuously  dynamic  system. 
There  are  a  large  number  of  connection  lines  among  neurons, 
forming  a  large  number  of  synaptic  connections.  Therefore,  this 
is  very  difficult  with  redoubled  efforts  but  limited  results, 
when  using  only  traditional  neuroscientific  methods  to  study  the 
computational  properties  of  the  bioneural  network.  On  the  other 
hand,  it  does  not  do  justice  to  reality  to  describe  in  detail, 
from  random  sampling  of  synaptic  connections  and  neuronal 
activity,  how  a  neural  network  computes:  what  sort  of  computation 
does  the  neural  network  do?  Based  on  Marr's  computational 
theory,  we  should  distinguish  between  the  abstract  theoretical 
level  and  the  real  neural  mechanism.  It  is  unrealistic  to  expect 
to  determine  the  algorithms  by  relying  only  on  low-level  neural 
mechanisms.  An  appropriate  method  is  to  start  from  a  theoretical 
viewpoint  to  establish  the  model  wiring  of  the  neural  network, 
and  t-~>  analyze  its  dynamic  properties  in  order  to  reveal  its 
biological  meaning. 
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In  reality,  as  early  as  more  than  40  years  ago  researchers 
began  to  follow  this  appropriate  method;  many  useful  results  [4] 
were  obtained.  Representative  examples  of  this  kind  of  early 
research  are  the  threshold  value  element  model  by  McCulloch  and 
Pitts,  the  linear  element  model  of  Hartline,  and  the  neural 
memory  model  of  Caianiello.  Recently,  extensive  studies  were 
conducted  on  the  doubled-valued  and  continuous  nonlinear  neuron 
model  circuit  when  exploring  the  problem  of  associative  memory. 
For  example,  as  pointed  out  by  Little  [5],  if  the  synaptic 
connections  in  a  neural  network  are  symmetric,  then  the  system 
will  evolve  to  a  fixed  sequential  state.  Professor  Hopfield  of 
the  California  Institute  of  Technology  introduced  the  concept  of 
energy  function  in  his  model  neural  network,  indicating  [6]  that 
the  system  will  be  eventually  in  a  state  of  minimum  energy 
function  if  the  synaptic  connections  are  efficient  and  symmetric. 
Consequently,  he  proposed  a  rational  framework  [7]  for 
understanding  the  computational  properties  of  a  neural  network. 
One  may  conclude  that  this  framework  is  an  important  result  with 
new  meaning  as  obtained  under  the  guidance  of  Marr's  theory 
because  the  framework  has  four  following  properties  that  are 
noteworthy : 

(1)  A  category  of  optimal  problems  can  be  naturally 
projected  into  the  neural  network.  Through  the  relationship 
between  the  system  stabilized  point  and  the  dynamic  process 
exposed  by  the  energy  function,  direct  understanding  can  be 
reached  on  how  a  neural  network  solves  this  kind  of  problem. 

(2)  This  relationship  confers  an  auxiliary  advantage:,  a 
contrast  can  be  made  with  spin  glass  with  its  large  amount  of 
data  on  statistical  mechanics,  thus  introducing  methods  of 
physics  and  systems  science  into  research  on  neuroscience. 

(3)  Concerning  methodology,  this  framework  has  its  roots  in 
the  theory  of  visual  computation,  therefore  the  framework  can  be 
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used  to  solve  some  problems  about  vision.  In  this  sense,  the 
framework  is  really  a  development  of  Marr's  computational  theory. 


(4)  In  nature,  the  computations  executed  by  the  model  neural 
network  differ  from  Boolean  algebraic  operations,  therefore  the 
model  has  very  great  appeal  to  researchers  on  new-generation 
computers  currently  under  development. 

The  paper  presents  Hopfield's  work,  stressing  the 
description  of  collective  computational  properties,  and 
explaining  that  his  proposed  rational  framework  is  a  natural 
method  of  studying  neural  network  computation.  Finally,  the 
authors  apply  the  model  neural  network  in  elementary  vision  in 
order  to  enable  readers  to  understand  and  evaluate  Hopfield's 
model  with  its  significance. 

II.  Model  Circuit  of  Neural  Network 

Let  us  consider  the  following  bioneuron  model:  its  input  is 
from  another  neuron  to  its  dendrite,  and  its  output  is  from  the 
axon  synapse  to  another  neuron.  The  action  potential  is 
generated  near  the  cell  body  and  is  conducted  along  the  axon  to 
stimulate  the  synapse.  Assume  that  the  electrical  effects  as 
caused  by  dendrite  shape  and  axon  can  be  neglected,  and  only  fast 
synaptic  events  are  considered,  then  the  potential  variation 
leads  to  simultaneous  variation  of  electrical  conduction  in 
cell  i  presynaptically  when  the  potential  variation  occurs  in 
cell  j.  The  magnitude  of  electric  conduction  variation  is 
determined  by  nature  and  intensity  of  the  synapse  from  cell  j  to 
i . 


The  neuron  generating  action  potential  operates  in  the 
steady  state  as  described  above;  its  impulse  transmission 
velocity  is  determined  by  input  current  from  the  synapse.  By 
changing  the  cell  body  and  cellular  current,  the  input  current 
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plays  its  role  indirectly.  The  discharge  time  constant  is 
determined  by  cell  capacitance  C-^  and  membrance  resistance  . 

By  integrating  time  constant  R^C^  over  input  current,  the  value 
of  the  equivalent  input  potential  u^  can  be  determined. 

Actually,  u-^  is  the  cell  membrane  potential  after  subtraction  of 
the  action  potential.  The  action  potential  is  the  reaction  of 
postsynaptic  cell  body  after  induction  of  the  synapse;  the 
reaction  velocity  is  determined  by  the  value  of  u^,  denoted  by 
fj^(u^).  The  relation  between  conduction  velocity  and  input 
current  (ujJ  can  be  described  with  a  simple  harmonic  S-shaped 
curve.  By  inputting  a  synapse  into  neuron  j  presynaptically  into 
neuron  i  that  is  postsynaptic,  the  synaptic  current  intensity  is 
proportional  to  the  product  of  fi(u^)  of  neuron  j,  and  the 
synaptic  intensity  T^j  from  j  to  i.  In  other  words,  current  of 
presynaptic  neuron  i  is  determined  by  T^j  x  fj(uj);  hence,  the 
function  of  the  action  potential  is  described  with  a  continuous 
variable. 


Many  neurons  operate  according  to  the  graded  response  mode; 
they  generally  do  not  generate  an  action  potential.  However,  the 
presynaptic  terminal  can  secrete  a  neurotransmitter,  hence 
capable  of  inducing  current  postsynaptically .  The  current 
generating  rate  is  determined  by  the  potential  of  the  presynaptic 
cell.  This  kind  of  neuron  equivalent  output  is  also  an  S-shaped 
simple  harmonic  function  inputted  by  the  synapse.  Hence,  whether 
an  action-potential-generating  neuron  or  a  hierarchical-reaction- 
causing  neuron,  their  model  neurons  can  be  described  by  using  the 
same  mathematical  form  as  shown  in  Figure  1A. 


Assume  that  a  neural  network  is  composed  of  N  neurons. 

Under  the  above  mentioned  hypothesis,  the  dynamic  equation  [6]  of 
neural  network  can  be  obtained; 
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dui 

'  dt 
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Fig.  l.  Input — Output  Relation  (A)  of  Model  Neuron 
and  Electron  Simulation  Circuitry  of  Model  Neural 
Network  (quoted  from  [6]). 

Key:  1.  Impulse  transmission  velocity  f(u); 

2.  Input  potential  u. 

This  nonlinear  differential  equation  set  can  be  considered  a 
kind  of  description  in  classical  nerve  dynamics.  It  shows  that 
to  neuron  i ,  the  net  input  current  of  u^  for  charging  input 
capacitance  is  composed  of  the  three  following  parts: 
a.  current  behind  the  synapse  induced  in  neuron  i  due  to  activity 
of  resynaptic  neuron  j;  b.  leakage  current  in  neuron  i  as  flowing 
past  the  input  resistance  R^ ;  and  c.  input  current  from  other 
neurons  extrinsic  to  the  network.  For  any  assumed  neural  network 
defined  by  specific  values  of  T^j,  1^,  f^,  C ^  and  R^ ,  its  time 
evolution  process  can  be  obtained  by  numerical  integration  of 
equation  (1). 

As  described  in  equation  (1),  the  system  dynamic  process  can 
also  be  simulated  with  an  electronic  circuit  shown  in  Figure  IB. 
This  feature  is  very  important  because  electronic  simulation  not 
only  can  present  a  direct  image  of  the  dynamic  process  of  the 
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network,  but  also  provide  a  direct  image  of  network  dynamic  state 
in  addition  to  providing  a  direct  avenue  for  realizing  the 
technique  of  network  computational  functioning.  In  this  model 
nerve  network,  a  neuron  is  a  computational  amplifier  with  a 
feedback  circuit  that  is  made  up  of  resistance,  capacitance  and 
connecting  wires,  capable  of  connecting  axon,  dendrite  and 
synapse.  Like  equation  (1),  the  input  current  of  any  amplifier 
is  composed  of  three  parts:  current,  leakage  current  and 
extrinsically  inputted  presynaptic  current.  Table  1  shows  the 
parametric  corresponding  relation  between  the  electronic  and  the 
model  neurons. 

In  the  model  expressed  in  equation  (1)  and  Figure  1, 
considerable  simplf ication  was  made  on  the  properties  of 
bioneurons  but  the  following  features  of  the  biological  system 
are  stressed  because  they  are  important  to  the  system: 

1.  For  the  neuron  as  an  input-output  conversion  apparatus, 
its  transmission  property  is  an  S-shaped  smooth  curve  from  0  to 
the  maximum  output. 

2.  The  cell  membrane  has  a  function  of  time  space  summation. 

3 .  There  are  large  numbers  of  excitatory  and  inhibitory 
connections  among  neurons;  these  kinds  of  connections  are 
realized  mainly  through  feedback. 

4.  By  representing  the  immediate  generation  of  the  action 
potential,  the  neuron  also  represents  the  ability  of  operating  in 
the  graded  response  mode . 

Therefore,  what  the  model  neuron  precisely  retains  is  the 
two  most  important  computational  (dynamic  and  nonlinear) 
features.  Undoubtedly,  this  is  an  appropriate  model  for 
theoretical  research  aimed  at  explaining  how  to  generate  the 
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intensive  computational  capability  by  coordinated  functioning 
among  neurons . 


Table  1.  Corresponding  Relationship  Between 
Amplifier  and  Model  Neuron  Parameter. 
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Key: 


1.  Name  of  parameter; 

2.  Model  neuron; 

3.  Amplifier; 

4.  Membrane  potential  of  input  neuron; 

5.  Output  (transmission  velocity  of  action 
potential  or  current  velocity  after 
causing  synapse); 

6.  Connection  strength  of  synapse; 

7.  Input  impedance; 

8.  Leakage  current; 

9.  Extrinsically  input  current; 

10.  Here; 

11.  Provided  by  connection  line  IN. 


is  the  output  potential  of  the  amplifier;  v^max  is  the 
proportionality  factor;  P j_(Uj_)  is  dimensionless  function  with  the 
same  shape  as  that  of  f^(u^).  The  maximum  output  values  of  the 
amplifier  is  1;  V|  is  the  corresponding  maximum  transmission 
velocity  of  neuron  i;  l/(Ri j )=!/( absolute  value  of  T^j)  is  the 
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feedback  resistance  of  the  amplifier;  by  connecting  with  the  plus 
phase  amplifier,  R^j  denotes  an  excitatory  synapse  (for  the  case 
of  linking  with  the  reverse-direction  amplifier,  R^j  denotes  an 
inhibitory  synapse);  and  IN  connects  with  a  direct  current  supply 
extrinsic  to  the  network. 

III.  Rational  Framework  for  Understanding  Dynamic  Features  of 
Neural  Network 

As  expressed  in  equation  set  (1)  and  Figure  1,  the  features 
of  the  neural  network  is  determined  by  synaptic  connection 
strength  T^j  and  extrinsically  input  current  1^.  For  a  given  T^j 
and  1^,  the  system  state  can  be  described  with  output  value  of 
various  neurons  in  the  network.  Assume  that  corresponds  to  a 
coordinate  axis  in  N-dimensional  Cartesian  coordinate  system, 
then  the  system's  instantaneous  state  can  be  expressed  with  an 
N-dimensional  vector  that  is  a  point  in  N-dimensional  space; 
hence,  the  dynamic  process  of  the  network  is  the  movement  of  this 
point  in  the  state  space.  The  computational  result  of  the 
network  is  the  steady  state  of  movement. 

Assume  that  the  system  has  several  locally  steady  limiting 
points  Xa,  Xv, ,  .  .  .  ,  then  the  computational  result  is  certainly  X;^ 
Xa  when  the  system  is  sufficiently  close  to  the  position  of  Xa  at 
the  start  of  processing  of  X=Xa+A.  In  this  state,  we  can 
consider  the  information  stored  in  the  system  as  vectors  Xa,  Xj-,, 
....  If  the  beginning  position  X=Xa+A  is  one  part  of  the 
information  of  Xa,  then  the  system  will  spontaneously  release 
complete  information  of  the  generating  Xa.  This  system  that  is 
capable  of  producing  complete  information  is  called  a  memory  with 
data  addressing,  generally  also  called  an  associative  memory 
device.  Obviously,  the  nature  of  associative  memory  processing 
is  that  the  system's  dynamic  process  should  be  convergent  to  a 
set  of  locally  steady  points;  the  other  kind  of  processing 
executed  by  the  neural  system  is  also  important,  as  we  will  see. 
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From  the  viewpoint  of  mechanics,  a  locally  steady  point 
corresponds  to  a  state  in  which  the  system  energy  is  at  a  locally 
minimum  state.  Hence,  the  neural  network  can  at  least  be  defined 
mathematically  as  a  computational  energy  function,  the  Liapunov 
function.  Through  the  function,  the  collective  computation 
feature  of  describing  a  neural  network  can  be  described.  For 
example,  the  associative  memory  network  can  be  defined  as  energy 
function  (6). 


Em 


£/,F, 


(2) 


In  the  equation,  gi-1(V^)=u^,  indicating  the  input-output 
relation  of  a  neuron.  Assume  that  Tjj  is  effectively  symmetrical 
(functioning  through  interneurons);  in  the  deriving  time 
derivative  from  equation  (2),  we  obtain 


dE 

dt 
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Substitute  equation  (1)  into  the  above  equation,  then  we  have 
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Since  gi_1(V^)  is  a  simple  harmonic  increasing  function,  C-^ 
is  a  positive  number;  hence,  no  negative  values  can  be  obtained 
in  various  summation  items  in  equation  (4).  Then  we  have 


dECQ  ii  -o.  v. 

dt  *0,  dt  dt 


(5) 


An  energy  function  always  has  limit;  therefore,  equation  (5) 
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indicates  that  the  system  in  state  space  can  enable  the  total 
movement  energy  find  th«  position  of  the  minimum  value  for  energy 
function  E,  and  stays  at  these  positions. 


Figure  2  shows  the  system's  energy  terrain  diagram  composed 
of  two  neurons  corresponding  to  two  steady  points  with  very  small 
energy  within  a  square  prescribed  by  two  steady  points  with  very 
small  energy  inside  a  square.  However,  when  the  amplifier  has 
very  high  gain  so  that  the  integrating  item  in  equation  (2) 
approaches  zero,  then  the  system's  steady  point  will  be  composed 
of  the  locally  very  small  value  of  energy  function 


E 


-\ZZT„r,r,-Zy.i. 


(2) 


The  steady  points  are  located  at  2N  vertex  angles  of  an 
N-dimensional  hypercube.  At  that  time,  the  steady  points  of  the 
continuous  value  model  network  will  directly  correspond  to  steady 
points  of  the  two-valued  model  network  [8-9]. 


Fig.  2.  Energy  Contour 
Terrain  Diagram  of  Double 
Steady  State  Model  Neural 
Network  (arrowheads  in  the 
diagram  indicate  motion 
direction  (cited  from  [6])). 


Hence,  whether  a  continuous-value  model  or  a  divergent- 
value  model,  all  energy  functions  can  indicate  the  internal 
connection  between  the  system  steady  point  and  dynamic  process. 
Based  on  this  relation,  we  can  compose  an  associative  memory 
network  [6,  7],  Actually,  only  by  considering  that  the 
associative  momory  problem  as  an  optimal  problem,  can  it  be 
related  to  the  solution  of  the  neural  network  through  the  energy 
function;  thus,  the  problem  of  how  the  neural  network  solves  the 
associative  memory  problem  can  be  directly  understood.  In  other 
words,  when  placing  the  steady  state  of  the  system  on  a  set  of 
special  memory  states,  this  purpose  can  be  achieved  only  by 
selecting  the  appropriate  parameters  T^j  and  1^  so  that  the 
network  is  in  a  situation  in  which  the  energy  function  of  the 
memory  state  is  locally  minimum.  For  example,  for  a  set  of  a 
total  of  m  memory  states  V6,S=1,2, . . . ,m  select  the  energy 
function 


.  •  • 

•  jo1  then  we  can  derive  t  -  £  v\ v‘  and  1^=0 
*  *-»  1*1 

As  mentioned  above,  the  method  for  understanding  the 
associative  memory  network  is  of  general  significance.  This 
method  can  be  used  to  solve  many  problems,  such  as  analog  to 
digital  conversion,  decomposition/decision  of  signals,  linear 
planning  [10],  and  the  salesman's  travel  route  [7].  In  certain 
conditions,  the  optimization  problem  can  be  naturally  connected 
with  computations  executed  by  the  neural  network  through  the 
energy  function.  For  a  system  of  symmetry  in  synaptic  intensity, 
we  have  the  following  theorem;  in  a  symmetric  network,  if  the 
input-output  relation  of  a  model  neuron  is  simple  harmonically 
limited,  and  the  variation  is  very  slow  for  the  input  current  (if 
it  exists)  extrinsic  to  the  network  during  the  network 
computation  process,  then  the  energy  function  is  always 
attenuated  with  time  in  the  dynamic  process  of  neural  network  as 
described  in  equation  (1).  In  other  words,  beginning  from  any 
initial  state,  the  system  will  move  along  the  descending 
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direction  of  energy  function.  After  arriving  at  the  local 
minimum,  the  system  stops  moving.  However,  if  the  synaptic 
intensity  is  not  symmetric,  there  is  no  unified  movement 
criterion  in  the  system.  The  system's  attractor  can  possibly  be 
the  steady  point  and  also  possibly  be  the  limit  ring,  and  even 
fuzziness  phenomena  may  appear. 


Fig.  3.  Dynamic  Process  (A)  and  Energy  Fig.  4.  Limit  Ring 

Terrain  Diagram  (B)  of  Symmetric  Network.  Abstractor  Possibly 

Appearing  in  Unsymme- 
tric  Neural  Network. 

The  above-mentioned  concept  can  be  explained  with  the  phase 
space  diagram.  Figure  3  shows  the  two-dimensional  phase  plane 
diagram.  Every  curve  in  A  corresponds  to  a  movement  possibly 
occurring  in  the  system;  arrowheads  indicate  the  directions  of 
movement.  Every  movement  locus  approaches  a  steady  point  and 
stops  there.  B  is  the  energy  terrain  diagram  of  A.  Figure  4  is 
an  example  of  the  possible  appearance  of  limit  rings  in  an 
unsymmetric  network. 

The  energy  function  is  the  overall  quantity  of  the  system; 
every  neuron  cannot  be  in  a  condition  of  awareness.  The  state  of 
various  neurons  is  determined  by  the  neuron's  equation  of 
movement  (1).  Through  the  energy  function,  we  can  naturally 
project  a  specific  computation  problem  onto  the  neural  network 


with  symmetric  connections.  Thus  we  can  directly  understand  the 
fashion  of  solving  problems  in  a  model  neural  network.  As 
general  methodology,  the  energy  function  also  provides  an 
important  means  for  neurophysiologists  to  understand  the 
computational  features  of  the  bioneural  network.  Obviously,  the 
energy  function  is  like  the  concept  of  entropy  in  statistical 
mechanics. 

IV.  The  Problem  of  Solving  Difficulties  by  Using  Model  Neural 
Network 

We  know  how  to  solve  problems  of  associative  memory  in  a 
model  neural  network  by  using  the  energy  function.  However,  the 
associative  memory  problem  is  only  a  simple  optimization  problem. 
With  respect  to  complex  problems  such  as  a  salesman's  travel 
route  (called  as  the  TSP  problem  in  the  following),  how  can  the 
situation  be  handled? 

The  TSP  problem  is  a  notably  difficult  problem  that  has  been 
studied  intensively.  The  problem  is  quite  clearcut:  it  is 
required  that  a  salesman  covers  n  cities  according  to  a  certain 
sequence.  In  his  itinerary,  he  only  visits  every  city  once,  and 
finally  he  returns  to  his  starting  point.  In  the  question,  he  is 
to  select  the  shortest  closed  route  among  many  possible  routes. 
Here,  the  total  length  of  a  route  is  determined  by  distances 
between  paired  cities.  This  is  a  so-called  combinatorial 
explosion  problem;  the  computational  volume  will  rapidly  increase 
with  increasing  n.  When  n  is  a  very  large  number,  solving  this 
problem  with  present-day  computers  is  difficult.  However,'  the 
above-mentioned  difficulty  is  easy  to  overcome  by  using  the 
neural  network.  Like  the  associative  memory  problem,  the  TSP 
problem  can  be  solved  through  the  three  following  steps  [7]. 

4.1  Mode  of  formulating  the  TSP  problem 
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The  solution  of  a  TSP  problem  with  n  cities  is  a  sequential 
table  composed  of  n  cities.  Assume  that  the  final  positions  of  n 
cities  are  determined  by  the  output  status  of  n  neurons .  Then  it 
is  required  that  in  the  problem  of  n  cities,  there  should  be  N=n2 
neurons  to  express  a  complete  route.  For  convenience,  the  output 
of  n2  neurons  is  arranged  in  a  matrix.  For  the  problem  of  five 
cities,  one  state  of  network  neurons  is  as  follows: 


1 

2 

— 

3 

4 

5 

A 

0 

1 

0 

0 

0 

B 

0 

0 

0 

1 

0 

C 

1 

0 

0 

0 

0 

D 

0 

0 

0 

0 

1 

E 

0 

0 

0 

0 

(6) 


Equation  (6)  is  called  a  transfer  matrix,  which  indicates  a 
route  such  as  the  follows:  a  salesman  begins  his  journey  at  city 
C;  his  first  destination  is  city  A,  ....  Finally,  the  salesman 
returns  to  city  C.  Obviously,  in  an  effective  route-indicating 
solution,  the  position  occupied  by  any  city  cannot  be  more  than 
one;  a  position  can  be  only  occupied  by  a  single  city.  In  other 
words,  in  describing  the  output  state  of  an  effective  route, 
there  can  only  be  one  "1”  output  in  a  row,  or  a  column;  all  the 
others  are  ”0."  For  decoding  any  transfer  matrix  composed  of 
output  values,  a  route  (the  one  solution  of  the  problem)  can  be 
obtained.  It  is  not  difficult  to  understand  that  there  are  a 
total  of  n!/2n  closed  routes  for  a  TSP  problem  of  n  cities. 

4 . 2  Energy  function  of  TSP  problem 

The  energy  function  of  describing  a  TSP  problem  should 
satisfy  the  two  following  requirements: 


(1)  The  energy  function  should  facilitate  reaching  the 
most  stable  state  for  the  transfer  matrix  expressed  by 
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equation  ( 6 ) ;  and 


(2)  The  energy  function  should  facilitate  expression  of  the 
shortest  distance  route  in  n!/2n  solutions. 


Considering  requirement  (1),  the  following  energy  function 
can  be  selected* 


£>  ‘  i  + f 11 1 ^  +  f ‘22 

I  a  4  ,  , 


(7) 


Considering  requirement  (2),  the  following  energy  function 
can  be  selected: 


(8) 


Numerically,  in  equation  (8)  E2  is  equal  to  the  length  of  several 
lines.  In  equations  (7)  and  (8),  the  outputs  of  the  neurons  are 
expressed  with  dual  footnotes.  The  row  footnotes  are  the  names 
of  cities  while  column  footnotes  are  the  positions  of  these 
cities.  To  express  the  final  effect  of  several  closed  lines,  the 
modulus  of  footnotes  is  n. 


The  overall  energy  function  is  the  summation  of  equations 
(7)  and  (8).  If  ABC  is  a  large  enough  number,  all  low  energy 
states  are  forms  of  expressing  effective  routes  for  the  network 
described  with  the  energy  function.  However,  the  state  with  the 
shortest  route  is  the  state  in  which  the  energy  is  at  the  lowest. 
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(2)  «ffSJ:iW{4£ 


Fig.  5.  Convergent  Process  of  Model 
Neural  Network  for  Solving  a  TSP 
Problem  With  10  Cities  (quote  [7]). 


Key:  (1)  City; 

(2)  Positions  at  route. 


4 . 3  Composing  a  model  neural  network 


Based  on  equations  (7)  and  (8),  the  joint  matrix  T-^j  can  be 
determined  by  secondary  terms  involving  the  energy  function; 
however,  the  extrinsic  input  current  1^  is  determined  by  the 
linear  terms.  By  the  dual  footnotes  method  as  mentioned  above, 
we  can  obtain  the  joint  matrix  defined  by  the  implicit  function 
as  follows: 


limiting  connection  with  each  row 
limiting  connection  within  each  column 
integrated  limiting  function 
data  term 


"  ‘o,  other  situations] 


The  externally  inputting  current  is 


(9) 


/  , +c,  stimulation  offset  (10) 
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In  T^j,  the  data  term  for  the  system  as  D  is  a  kind  of 
input;  this  explains  which  TSP  problem  should  be  solved. 

However,  as  required  by  any  TSP  problem,  a  general  constraint  is 
provided  for  those  terms  with  coefficients  as  A,  B  and  C.  Since 
the  energy  function  controls  the  dynamic  process  of  the  network, 
the  final  state  like  that  in  equation  (6)  can  be  derived  after 
beginning  with  the  initial  offset  of  the  network.  In  other 
words,  the  network  can  select  the  optimal  circuit  among  n!/2n 
final  states  satisfying  the  constraint  conditions  under  the 
function  of  data  terms. 

Figure  5  shows  the  computer  simulation  result  of  a  TSP 
problem  with  10  cities.  The  network  never  inclines  toward  any 
special  circuit  of  its  state  with  a  noise  function  for  beginning 
its  operation.  After  a  very  short  time  period  (approximately 
several  time  constants  for  states  a,  b  and  c),  the  network  is 
convergent  at  state  a.  For  the  problem  of  10  cities,  there  are  a 
total  of  101/20  (approximately  2  x  105)  circuits.  From  this  many 
circuits,  the  network  selects  the  two  shortest  circuits. 
Therefore,  the  selectivity  coefficient  of  this  sort  of  network  is 
2/(2  x  105)  =  10-5. 

For  a  TSP  problem  with  30  cities,  the  total  number  of 
circuits  is  approximately  1030.  Therefore  this  is  a  difficult 
(to  control)  problem.  However,  in  a  one-time  convergence 
(approximately  several  time  constants),  the  network  composed  of 
30x30  neurons  can  select  107  good  solutions  in  expelling  bad 
solutions  numbering  1023  times  (good  solutions).  The  selectivity 
coefficient  is  10-23. 

Thus,  it  can  be  seen  what  a  good  approach  in  solving  the  TSP 
problem  we  have  been  examining.  We  have  reason  to  believe  that 
this  unusually  rapid  and  effective  computation  capability  is  a 
natural  result  of  utilizing  the  continuous  variant  and  feedback 
loop.  System  0-1  does  not  have  this  sort  of  capability.  Hence, 
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the  computation  executed  by  model  neural  network  is  concretely 
different  from  the  Bohr  logic  operation. 

V.  Application  of  Model  Neural  Network  in  Initial  Stage  Vision 

As  described  above,  the  Hopfield  model  has  been  developed 
under  guidance  of  the  theory  of  computer  vision;  therefore,  the 
model  naturally  can  solve  problems  in  vision  research.  For 
example,  we  can  apply  the  model  to  solve  the  surface 
interpolation  problem  [11]  capable  of  maintaining  noncontinuity. 

There  are  two  methods  of  reestablishing  the  surface  from 
data  measurement:  one  is  the  reestablishment  of  a  smooth  surface, 
capable  of  being  solving  with  the  standard  regularization  theory 
[12].  Another  method  is  the  reestablishment  of  a  smooth  surface 
section  by  section;  this  is  a  problem  of  nonsecondary  type, 
capable  of  being  solved  with  random  regularization  method  [13]. 
The  model  of  reestablishment  of  a  smooth  surface  section  by 
section  is  composed  of  two  mutually  coupling  Markov  random  field: 
one  is  the  field  of  taking  continuous  values;  the  depth  value  is 
fj_  as  measured  corresponding  to  position  i.  Another  is  the  field 
of  taking  secondary  values;  its  variant  is  between  lattice  points 
of  depth  measurement.  In  this  implicit  function  of  linear 
treatment,  whether  noncontinuity  exists  between  two  adjacent 
depth  elements  is  to  be  pointed  out.  Under  the  Bayesian  estimate 
corresponding  to  an  energy  function,  the  overall  property  of  the 
maximum  estimate  of  the  surface  is  very  small. 

As  pointed  out  in  the  Hopfield  model,  the  relationship  in 
state  variants  between  the  continuous-valued  model  and  binary¬ 
valued  model  is  determined  by  the  gain  of  the  amplifier.  When 
x-yoo  ,  the  amplifier  output  is  0  or  1.  Hence,  the  linear 
treatment  of  the  two  values  can  be  projected  in  continuous 
variant  limited  by  0  or  1.  By  only  selecting  the  appropriate 
energy  function  and  renewal  rule,  it  can  be  proven  that  the  total 
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energy  of  the  network  is  always  attenuated  with  time.  The  system 
will  evolve  in  attenuation  fashion,  and  finally  a  minimum  value 
of  E  is  attained.  To  a  diagram  composed  of  32x32  image  elements, 
the  result  of  computer  simulation  is  shown  as  in  Figure  6.  The 
left  upper  corner  in  the  figure  shows  the  initial  state  of  the 
network;  the  right  lower  corner  shows  the  final  state.  Thus,  it 
is  effective  by  using  the  Hopfield  model  in  reestablishment  of 
the  smooth  surface  section  by  section. 

Figure  7A  is  a  simulation  network  of  carrying  out  surface 
smooth  interpolation  section  by  section.  The  basic  concept  is  to 
combine  a  programmable  parallel-processing  device  with  local 
connection,  and  a  linear  network  composed  of  electric  resistance 
in  a  hybrid  device.  Corresponding  to  the  linear  element,  a 
digital  processor  can  cut  off  the  resistance  connection  between 
two  adjacent  simulation  processors  by  using  simple  switches. 

This  kind  of  hybrid  device  has  two  periods  of  fundamental 
operations.  Within  the  period  of  simulation  operation,  the 
processor  leads  the  current  (corresponding  to  depth  measurement) 
into  a  simulation  network.  Then  under  the  given  linear 
distributed  situation,  the  resistance  network  finds  out  the 
(only)  surface  as  the  smoothest.  Within  the  period  of  digital 
operation,  the  digital  processing  network  reads  out  the  current 
values  of  resistance  at  various  nodal  points  in  the  resistance 
network.  In  addition,  by  using  the  software  of  a  compiled 
program,  recomputation  is  conducted  with  linear  processing  of  the 
two  values.  Here,  the  random  optimization  method  is  easily 
carried  out.  Then  the  processing  will  appropriately  break  off 
connection  lines  in  the  simulation  network.  In  designing  an 
artificial  vision  system,  this  configuration  of  the  computation 
network  is  a  new  concept.  Figure  7B  shows  an  imaginary  mechanism 
of  the  neural  network  of  the  hybrid  device. 


1.6  6.4 


Fig.  6.  Processing  Procedure  of 
Neural  Network  (quoted  from  [11]) 
for  Interpolation  of  Field  Scene 
Composed  of  Three  Inclined  Cuboids. 


Fig.  7.  Mixed  Type  Circuit  (A)  and  Its  Nerve 
Realization  (quoted  from  [11])  in  Carrying  out 
Smooth  Surface  Interpolation  Section  by  Section. 


VI.  Discussion 

After  we  have  described  the  general  situation  of  neural 
network  research  and  the  rational  framework  of  collective 
computation  of  neural  network,  the  following  conclusions  can  be 
reached: 
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(1)  Computer  vision  reveals  the  prologue  of  computational 
neuroscience.  Computer  vision  can  be  regarded  as  a  genuinely 
borderland  discipline;  with  its  lead,  computer  science, 
mathematics,  physics,  systems  science  and  systems  engineering  (in 
addition  to  the  traditional  neuroscience  method)  are  included  in 
the  realm  of  neuroscience,  which  begins  to  show  its  brightness. 
This  method  of  utilizing  interdisciplinary  research  is  an 
important  source  of  creative  concepts.  Undoubtedly,  this 
approach  will  considerably  speed  up  neuroscience  research  in 
markedly  changing  the  achievements  of  neuroscience.  Recently,  a 
Department  of  Brain  and  Recognition  Science  was  established  at 
the  Massachusettts  Institute  of  Technology,  and  again  recently  a 
Research  Project  of  Computation  and  Neural  Systems  was  proposed 
by  the  California  Institute  of  Technology.  These  are  important 
decision  steps  taken  with  this  kind  of  recognition.  Hence,  we 
have  reason  to  believe  that  computational  neuroscience  has  become 
a  major  research  direction  in  neuroscience. 

(2)  In  the  rational  framework  of  Hopfield's  collective 
computation  of  a  neural  network,  a  category  of  the  optimal 
problems  can  be  naturally  projected  into  a  neural  network  with 
symmetric  connections.  In  particular,  with  the  introduction  of 
the  energy  function  and  the  application  of  design  circuitry,  we 
can  understand  how  the  optimal  solution  can  be  derived  by  a  model 
neural  network,  which  realizes  the  features  of  analog  and  digital 
computers.  However,  essentially  the  model  neural  network  is 
different  from  either  of  these  computers.  On  the  one  hand,  the 
network  integrates  data  expression  and  programming  of  digital 
computation.  On  the  other,  the  network  uses  analog  computation 
to  replace  Boolean  logic  operation.  Here,  the  differential 
equation  of  the  model  is  not  of  concrete  significance; 
essentially,  the  differential  equation  is  a  program;  through  it, 

a  solution  of  the  problem  can  be  obtained.  This  sort  of 
collective  computation  property  of  the  model  neural  network  is 
obviously  very  significant  to  research  on  new-generation 
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computers  under  development  or  present-day  intelligent  computer. 
Actually,  Hopfield's  work  attracted  the  general  attention  of 
researchers.  Not  only  are  there  high  challenges  at  the  level  of 
theoretical  research,  but  the  corresponding  chips  were  also 
manufactured.  For  example,  chips  for  solving  movement  computing 
problems  have  been  manufactured  in  the  United  States.  Therefore, 
we  should  stress  this  new  trend  of  neuroscience  research. 

(3)  At  the  same  time,  we  must  realize  that  the  present 
results  have  considerable  limitations  because  research  in 
computational  neuroscience  is  in  its  infancy.  Many  problems  are 
waiting  for  solution  by  researchers. 

For  example,  network  parameters  are  given  in  Hopfield's 
theoretical  framework.  In  other  words,  his  proposed  parameters 
of  a  model  neural  network  are  written  into  the  network  after  man 
have  advanced  in  computer  science.  The  network  by  itself  does 
not  have  properties  of  learning  and  adaptation.  Hence,  it  should 
be  clarified  how  a  neural  network  with  certain  functions  can  be 
self-organized  [14]  through  learning  and  adaptation  in  order  to 
achieve  the  unity  of  human  intelligent  behavior  and  neural 
structure;  there  is  a  relatively  long  distance  to  be  covered. 

For  another  example,  Hopfield's  model  demands  that  the 
connections  of  a  neural  network  should  be  symmetric.  This  point, 
however,  is  not  established  in  all  situations.  Actually,  some 
networks  are  not  symmetrical.  One  of  the  dynamic  features  of 
unsymmetrical  network  is  the  existence  of  limit  ring  attractor, 
whose  concrete  computational  approach  and  meaning  are  not 
understood  by  researchers,  not  to  mention  fuzziness  phenomena. 

For  this  kind  of  theoretical  problems,  it  is  possibly  required  to 
have  even  higher  theoretical  viewpoints  for  research  treatment. 

On  this  aspect,  some  help  to  research  can  be  provided  by 
Prigogine's  theory  on  dissipation  structure,  Haken's  coordination 
theory  theory,  and  Thom's  mutation  theory.  It  can  be  believed 
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that  once  theories  like  dissipation  structure  are  established  in 
neuroscience,  this  will  a  new  face  in  a  revolutionary  change. 


The  focus  of  actually  applying  Hopfield's  model  is 
concentrated  at  present  on  processing  problems  of  associative 
memory  and  initial  stage  vision.  According  to  Hopfield's 
estimate  [8],  the  model's  memory  capacity  is  only  N=0.15n. 
However,  if  signals  are  appropriately  encoded  in  advance,  memory 
capacity  can  be  greatly  increased.  Hence,  during  research  on 
problems  of  associative  memory,  research  on  programming 
technology  should  be  opportunely  developed.  Hopfield's  model  has 
some  use  in  initial-stage  vision  as  the  model  can  solve  problems 
[11]  that  cannot  be  solved  by  the  standard  regularization  theory. 
However,  when  the  energy  function  is  of  non-secondary  type,  at 
present  it  cannot  ensure  that  the  network  can  process  to  the 
state  in  which  energy  is  at  the  minimum.  Development  and 
perfection  of  this  aspect  of  work  is  also  a  problem  concerning  by 
researchers . 

The  paper  was  received  for  publication  on  25  July  1987. 
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ORGANIZATION 


MICROFICHE 


C509  BALLISTIC  RES  LAB  1 
C510  R&T  IABS/AVEADCTM  1 
C513  AKRADCCM  1 
C535  AVRAK3JVTSAROCM  1 
C539  IRASANA  1 
C591  FSTC  4 
C619  MIA  REDSTONE  1 
D008  MISC  1 
E053  USAF/INET  1 
E404  AEDC/DOF  1 
E408  AFWL  1 
E410  AD/XND  1 
F429  SD/IND  1 
P005  DOE/ISfl/DDI  1 
P050  dA/OOVADD/SD  2 
AFTT/ITE  1 
NOIC/OIC-9  1 
OCV  1 
MLA/FHS  1 
UM/CDCE  L-309  1 
NASA/NSP-44  1 
NSA/T513/TDL  2 
ASD/FTD/TTIA  1 
ESL  1 
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